How reliable is the claim that sugary drinks

are killing 184,000 people every year?

Sugar-Sweetened Beverages (SSBs)—soda, fruit juice, iced tea to the non-academic—have increasingly been blamed for fattening and sickening the world; now, according to a new study from researchers at Tufts University, their consumption kills 184,000 people across the globe each year.

The drink of death makes for irresistible headlines, from the conventional— “The US leads the world in soda-related deaths, but poor countries suffer the most”—to the imaginative— “Study estimates sugary drinks more deadly than violent crime in Mexico.”

We have photographic evidence and eyewitness accounts of deadly crime in Mexico, but where exactly were the bodies buried in the model that produced these numbers? Unfortunately, the way this study was reported in the media suggests that journalists prefer drinking Kool Aid to applying skepticism—and there are many good reasons for skepticism about this study, given its limitations.

Of special concern is the uncertainty of the numbers due to a variety of factors, from how the estimates of consumption were made to how the authors drew conclusions about death rates based on the consumption estimates. The 184,000 estimate is the product of a model and the step-by-step decisions made by the researchers as to what to put into their model. Each decision at each step affects the other steps. Unfortunately, the researchers in this study were unclear regarding several choices that could massively impact the outcome. The problems include:

Major Sampling problems

• Estimates (derived from models) about the SSB consumption in many countries for which no survey data exists. This results in rather surprising claims, such as the idea that Afghani women ages 20-44 consume 0.4 servings of SSBs per day (even though South Asia was not sampled). This model estimate is the same for women of the same age in Albania, a world away in culture, and much greater than the estimate for women of the same age in Italy (0.2 servings per day).
• Unrepresentative (or no) sampling for a large percentage of the world population. Only 51 countries (of 187 predicted from the model) had survey data available.
• Surveys were not generally representative. How would 22,000 Australians be representative of 19 million people (0.1 percent) or 71,000 North Americans representative of 244 million people (.02 percent), or 1,500 people surveyed from sub-Saharan Africa be fully representative of 400 million people (0.00038 percent), given the racial, ethnic, and economic diversity across these continents?
• Reliance on self-reported data known to be problematic.
• Some of the surveys were conducted many years ago; the only survey data they report from sub-Saharan Africa is 20 years old.

Missing data

• No reports of the percentage of missing data from the surveys, and how it is addressed by the models or otherwise. Missing data (or nonresponse) can create tremendous bias in a survey; if, for example, people who are healthier are also less likely to complete the survey, then we may have artificially inflated health problems associated with soda consumption. Not knowing what the authors did in the models makes it difficult to ascertain whether corresponding bias is a concern and whether the authors account for it appropriately.

Proxy measures

• The use of the availability of sugar according to the United Nations Food and Agriculture Organization (UN FAO) in the models as a means to estimate SSB consumption but with no apparent adjustment for food waste or the use of sugar in other foods than SSBs. Presumably this data was most helpful for the countries with no other data on consumption.

Time

• No description of how time is modeled. While the study claims to describe “SSB Consumption in 2010,” the authors model the consumption in 2010 based on data collected many years prior. But are they assuming that increases in consumption over the years are linear? Or that consumption has increased exponentially? And how did they evaluate goodness of fit?

But even if one were to accept the consumption rates of SSB, we still run into other issues with how SSB consumption leads to death. These include:

• Lack of transparent use of uncertainty; it is unclear whether the authors appropriately accounted for the uncertainty in all of their estimates.
• No adjustment for sugar-containing foods (rather than SSBs) that could lead to similar outcomes as that attributed to SSBs. Or, to the extent that adjustments were made, they were unclear, unspecified, or inconsistent.
• Misplaced assumptions about who drinks SSBs and who dies from diseases.

The news media made no attempt to ask how the study produced this estimate even though the model lacked the kind of data it needed to support its claim.

Estimates of Sugar-Sweetened Beverage Intake
Despite media reports, the 2010 Sugar-Sweetened Beverage consumption was not based exclusively on survey data from 2010. Perhaps the confusion stemmed from the study’s authors words in the abstract:

“We modeled global, regional, and national burdens of disease associated with SSB consumption by age/sex in 2010” and “Disease-specific mortality/morbidity data were obtained from Global Burden of Diseases, Injuries and Risk Factors 2010 Study.”

It may be common parlance to refer to a model’s results for 2010 consumption as “2010 Consumption,” but it drops any hint that the data feeding the model were collected between 1981 and 2009, and relied in part on sugar availability reports for the United Nations Food and Agriculture Organization.

Yes, that is correct: the consumption of sugary drinks was based, in part, on the availability of sugar in that country. But the availability of sugar serves as a poor surrogate for the consumption of soda and other sugary drinks for two reasons: first, no one knows how much sugar was wasted. In the U.S., for example, estimates put food waste at 40 percent. For third-world countries, the amount is presumably far less, but food distribution can control food access—and it may be that food is being stockpiled rather than consumed (as is also done in the U.S.).

Just as egregious, however, is the assumption that sugar availability directly translates into sugary beverages. Candy, sweets, cereals, and many processed foods include sugar, making SSBs only one of several vectors for sugar consumption. And what about the survey data? According to the authors of the study, “these surveys provided data on SSB intake in countries representing 63 percent of the world’s adult population.”

In other words, more than a third of the world’s adult population live in countries that were not even surveyed. An important distinction: The authors did not claim to have a representative sample of 63 percent of the world, but rather that the 51 countries for which there are surveys at all cover 63 percent of the world population. Among the countries that were surveyed, the surveys were not necessarily representative of this 63 percent of people. And some of the surveys are more than a generation old, dating back to 1981.

For the moment, let us assume that the surveys are perfectly representative of their respective populations (which is next to impossible) and perfectly accurate reflections of SSB consumption (which is almost certainly untrue). These surveys are still unrepresentative of the world population. To illustrate, consider Afghanistan, a country for which no surveys on SSB consumption have been conducted. The study reports estimates of Afghani consumption of SSBs provided by the models, and the estimates say that young Afghani women are drinking about twice as much soda or other SSBs as Italian women of the same age, which is an unlikely conclusion. How exactly would one model the soda drinking occurring in war-torn and impoverished countries without actually asking people? By sugar availability? These wacky conclusions come from a vaguely described model on SSB consumption. As the authors of the study put it:

“To combine individual-level intake data with country-level food availability data, to address issues of data incomparability, and to capture the uncertainty in estimates of beverage intake due to measurement error, sampling uncertainty, and modeling uncertainty, we used established age-integrating Bayesian hierarchical modeling methods.”

The idea of Bayesian modeling is to make some reasonable assumptions about some behaviors, and to let these assumptions inform the calculations. The problem here is that the authors haven’t provided adequate information in their study to recreate their model and evaluate its appropriateness. These models “magically” find (at times, implausible) rates of consumption to input into other models to estimate death rates due to soda. One simply cannot get something (statistically speaking) from nothing (no data).

How do sugar-sweetened beverages translate into death?
The authors estimated the impact from SSBs on death rates as follows:

• Assume sugar-sweetened beverages directly cause diabetes and increased body mass index (BMI); derive estimates of the proportion of each caused by the SSB consumption model results.
• Assume elevated BMI causes cardiovascular disease (CVD), diabetes and cancer; use modeling to quantify the proportion of each caused by increased BMI—in turn, determine the amount attributable to SSB consumption estimates made previously.

But here’s where the uncertainty comes in:

• Uncertainty in SSB consumption.
• Uncertainty in the proportion of disease caused by this consumption.
• Uncertainty in the proportion of deaths caused by these diseases.

The authors state they have taken into account the uncertainty in SSB consumption and in the relative risk associated with the diseases, but they do not mention the uncertainty in the proportion attributable to SSBs. Sampling data (and models0 can never given an exact measure of SSB consumption, which is why the authors present a range of quantities that they believe are consumed; likewise, there is not enough data to say with certainty the exact percentage of diabetes cases caused by a particular level of SSB consumption.

The study does not mention how the latter uncertainty is taken into account; uncertainties propagate through calculations, so neglecting to account for uncertainty results in overly conclusive ranges of disease caused by SSB consumption. Just as importantly, it is crucial to define models that do not assume SSBs have a uniform impact on the population; the fallacy in the reasoning may be best illustrated with an imaginary country. In this country, wealthy people drink a lot of soda, but they have excellent health care, which helps them manage chronic diseases such as diabetes and CVD. These wealthy people are both the SSB consumers and the health care consumers, with little to no impact of SSBs on their death rates because of their excellent health care.

Poor people in this imaginary country, by contrast, do not consume SSBs nor do they have good health care. If they get diabetes, they die—yet the cause is not SSBs since they don’t drink them. The result? A country with high SSBs and high death due to diabetes, and yet the two are unrelated phenomena. In particular, lowering SSB consumption will not impact the death rates.

It is unclear whether the authors had sufficient data and appropriate models to account for this kind of non-uniformity in a real country with economically, culturally and biologically diverse populations. Many studies have shown an association between chronic diseases (and high BMI) and the consumption of soda. But it’s never clear if the soda is the problem specifically, or if the same people are also eating foods the have the same effect. Little evidence exists to point to soda alone, as opposed to an array of other nutritionally bereft foods consumed in excess. Eric Rimm, professor at Harvard School of Public Health, notes that “in some cross-sectional epidemiologic studies, obesity is associated with more diet soda consumption” (emphasis added).

This kind of observation makes conclusions about SSBs difficult—are SSBs what put people at risk, or other foods/behaviors/activities that people who consume SSBs engage in? Why might diet soda be associated with increased BMI (and therefore increased death)? Several explanations have been offered, including publication bias—the studies showing associations are more like to be published—and the fact that diet soda drinkers have poorer diets than those who drink no soda. Still others argue that diet sodas are at fault, because they lead consumers to other poor food choices.

The association, however, drives home the point that the amount of BMI increase associated specifically with sugar-sweetened beverages is difficult to pinpoint. A lot of consideration has been given to the increased caloric intake for people who drink SSBs. If SSBs are to be blamed for an increased BMI, then one would want to know that the products themselves—rather than their calories—lead to increased BMI. For example, bananas have about the same calories as a can of soda, yet people who eat bananas (even several a day) tend to reduce their caloric intake from other products. In contrast, people who drink soda tend to increase their calories, leading to weight gain.

But if unhealthy levels of caloric intake result from fries, cream-filled coffee, cheese galore, and soda, why should we blame the SSBs specifically for the problem? This issue relates to whether SSBs satiate hunger as well as other foods with similar caloric intake. If drinking a 150-calorie soda results in a person eating 150 calories fewer of potato chips (or broccoli), perhaps there’s no blame to place on soda as opposed to potato chips. If, in contrast, the soda does not have the same effect on one’s hunger as other foods might have—so that drinking a 150 calorie soda leads to one eating only 50 calories fewer at the next meal—then we can blame the increased BMI squarely on the soda. There is evidence that liquid calories don’t sate hunger the way solid food calories do; however, the increased liquid calories occur in tandem with other food increases, leading yet again to questions about which is to blame. Remember the potato?

For this study, the estimates of risk associated with consuming SSBs or with being overweight come from large studies across North America, Europe, and East Asia. The authors do not specify whether they accounted for the other foods people were eating in these studies. There is strong evidence that food intake (and a high sugar diet) can lead to all the diseases mentioned, but less evidence that SSBs are the main culprit rather than a part of a mosaic of unhealthy behaviors.

The overall conclusion? Models predicting world consumption of SSBs require real data that hasn’t been collected. Modeling the resulting world deaths without that data is fraught with uncertainty, which should not be hidden by dangling the lure (but not the details) of complicated models. This study is an acute reminder to journalists of the importance of asking where the data feeding a model comes from: No model on human behavior can accurately account for non-existent or severely unrepresentative data.

Please note that this is a forum for statisticians and mathematicians to critically evaluate the design and statistical methods used in studies. The subjects (products, procedures, treatments, etc.) of the studies being evaluated are neither endorsed nor rejected by Sense About Science USA. We encourage readers to use these articles as a starting point to discuss better study design and statistical analysis. While we strive for factual accuracy in these posts, they should not be considered journalistic works, but rather pieces of academic writing.