Glaring Flaws in Sugar Toxicity Study
Anew study has claimed that obese children could find rapid health improvement by small sugar reductions, without caloric restrictions. According to the lead author, Robert Lustig, the new study shows that sugar may not be harmful because of how it leads to weight gain, but “rather sugar is metabolically harmful because it’s sugar.” According to the study, a diet with 10 percent sugar in place of one with 28 percent sugar can in just nine days produce a reduction in blood pressure, triglycerides and LDL-cholesterol—and improved glucose tolerance and lower levels of insulin circulating in the blood.
Does a miracle diet promising incredible results in just nine days sound too good to be true? Not to the news media, which gobbled up the study’s conclusions as the proof that sugar really is the big evil in our diet (see sidebar). But as we shall see, the science in the study is about as good as it is for other fad diets.
“In a paper published Tuesday, he and his colleagues believe they have come up with the definitive evidence that sugar, as Lustig says, is toxic,” reports Time Magazine. “Up until now, there have been a lot of correlation studies linking sugar and metabolic syndrome,” says Lustig. “This is causation.”
In a story “Cutting Sugar Improves Children’s Health in Just 10 Days,” The New York Times turned to clinician Frank Hu of Harvard’s School of Public Health for comment, who told the paper that the study “strengthens the existing evidence on the relationship between added sugar intake and metabolic disease… This kind of study is very difficult to do,” he said. “But it provides a proof of concept that in a high risk population, reducing consumption of added sugar can have multiple metabolic benefits.”
Dr. Sonia Caprio, a professor of pediatrics at Yale Medical School also told the paper that “[I]t addressed the issue in an original way and tried to isolate the effect of sugar on metabolic syndrome and insulin resistance… This is an important area of research that might solve some of the metabolic issues that we are facing in children, particularly in adolescents.”
In The Wall Street Journal’s story titled “Study Links Sugar to Conditions That Lead to Diabetes, Heart Disease in Children,” Dr. David Ludwig of Boston Children’s Hospital, said “Because of the study’s design, we can’t be completely certain that the changes are fully attributable to changes in sugar intake… It’s possible that other aspects of the diet or lifestyle changed.” The study, he told the paper, was “an interesting and useful step forward in assessing the effects of added sugar in children.”
Hu and Ludwig are frequently quoted in stories about the dangers of sugar.
Both papers had quotes criticizing the study from food industry sources (albeit not very insightful quotes), as did the San Francisco Chronicle’s story and Deadspin’s summary of the findings. New York Magazine offered no critical reporting or analysis and neither did NBC News.
The Daily Mail, which quoted the paper’s senior author, Dr Jean-Marc Schwarz, of Touro University saying “I have never seen results as striking or significant in our human studies,” also offered no critical perspective.
The only newspaper which provided some genuine critical insight was The Guardian (which is particularly notable as it simultaneously ran a comment is free piece by Lustig titled “The science is in: the case for a sugar tax is overwhelming”):
“The results are not convincing to me – this is a very small study, and it has not been statistically well-controlled,” said Naveed Sattar, professor of metabolic medicine at Glasgow University. “Also, when people are losing weight, even if modest, their metabolic changes can seem larger than they actually are – one needs to see results once folk return to their habitual state after they’ve finished losing weight. Overall, this study is of modest interest but is far from convincing.”
Tom Sanders, professor emeritus of nutrition and dietetics at King’s College London, said the study needed to be viewed “with some skepticism” because it was uncontrolled. It did not compare the children with a similar group who continued to eat a high-sugar diet. The comparison instead was made with their weight and health before the study while on their usual diet. “But it is well known that obese children underestimate and under-report food intake, particularly of soft drinks and snack foods,” said Sanders.
“This is a fundamental flaw in the study. It is likely that the changes in metabolic outcomes observed can be explained by the experimental diet providing fewer calories than the children’s usual intake.”
That many doctors commenting on the study missed this—and the other significant problems with the study—is a cause for concern. How many commenting on critical public health issues do not know that control groups and correcting for multiple testing are basic principles of good study design? Contrast all this coverage with the critical analysis by Cornell nutrition PhD student Kevin Klatt.
TB
What the study attempted to do
Forty-three obese kids participated in the study, in which they were provided with nine days of food containing reduced sugar, and a number of medical tests were conducted on them. The boys and girls were aged between eight and 18-years old, with the average age being 13 years old. As might be expected, they exhibited a wide range of levels of pubertal development.
This kind of dietary research is always challenging. Observational studies on sugar consumption are rife with confounding factors: kids who eat more sugar have so many other differences compared to kids who eat less sugar, making it impossible to know if sugar is the culprit.
The Lustig study aimed, in contrast, to create a structured environment, in which obese children were given the same food they normally eat, except that most of the sugar was substituted by starch. The provided diet had the same number of calories, in theory eliminating an outcome that would be linked to reduced calories rather than reduced sugar. The food the kids ate was meant to preserve their weight and mimic their normal food intake, based on their self-reports of their current diet. Results from food surveys are notoriously incorrect. And as we will see, this little feature of the study may have an enormous impact on the results.
Controls. This is not a randomized experiment. The study included only kids given a special diet with no control group. Such a design can be extremely suggestive: it is certainly possible that the new reduced-sugar diet resulted in all of the positive effects seen by the researchers. But it is easy to see why, without controls, the results are difficult to interpret. The food choices used in the study were based on reports of what the kids normally eat—but we can’t know whether we have accurate data on what they normally eat. The kids spent 10 days thinking about what they eat (because they were provided with this special food) and weighing themselves daily. Maybe the mindfulness alone led to changes in eating habits, resulting in better biological metrics. Without knowing to what we are comparing the low-sugar diet, we cannot be confident that reduced sugar is the only difference from the original diet.
Perhaps a similar diet as the one provided, but with the amount of sugar reported by the kids, would also have resulted in reductions of blood pressure, and improved cholesterol. The study would have been far more powerful in answering its own question if it had provided another set of 43 kids a box of similar foods with their accustomed levels of sugars, parallel to the boxes provided the low-sugar kids. In addition to comparing each kid to his or herself over time, one could then compare the two groups and test whether improvements were more substantial for the low sugar group.
Weight Loss. The study grossly underestimated the number of calories required by the kids to maintain their weight—an error in calculation or through reliance on self-reported diets—but also one that reflects poor study design. The kids’ dietary needs should have been tested from the start. In theory, the kids were given enough food that they would not lose weight; yet, they lost an average of two pounds—and some of the kids lost much more—in just 10 days. The authors admit that they noticed very significant weight loss (more than two percent of body weight) in seven of the first 17 participating kids, and then increased the caloric recommendations at that time.
This feature of the study is important: if sugar is the cause of health problems, independent of all else, then the ideal experimental design would involve cutting out sugar while kids maintain their weight. Yet in this study, more than three fourths of the kids said they could not eat all the revised target amount of food to maintain their weight.
Results. Kids were given oral glucose tolerance testing (OGTT) on Days 0 and 10. This test requires fasting before administration; a sugary drink is given to the kids and then blood is drawn at minutes 0, 30, 60, 90 and 120 minutes after the kids finish the drink. The glucose level observed can be predictive of diabetes. There were differences in the glucose response from Day 0 to Day 10.
Yet it was unclear whether these differences would be due to the decreased sugar intake. Since 77 percent of the kids also lost weight, we might look to the ones who didn’t lose weight.
The authors claim that the ten kids whose weight remained constant also showed better health at Day 10 compared to Day 0; but here the data are far from clear. The glucose levels at minutes 0, 30, 60, 90 and 120 minutes after ingesting a sugar drink are extremely similar at Day 0 and in Day 10, whereas in the group that lost weight, the glucose levels are significantly higher Day 0 than Day 10. In that group of only ten kids, the peak glucose level was higher before the diet, but the subsequent measurements throughout the 120 minutes did not improve significantly, leaving doubt about the measured benefit. The differences in insulin levels from Day 0 and Day 10 among the ten kids who did not lose weight show no statistically significant differences.
Furthermore, the post hoc analysis of a subgroup is always problematic: are these kids different from the kids in the group of those who lost weight in other ways than just the fact that they didn’t lose weight? Their results may not reflect what would have happened had the 43 original kids all maintained their weight.
Finally, the collection of data may have been flawed by design. The kids may have been more primed to fast as they were supposed to for Day 10, after nine days of focus on eating and weighing themselves, than they were to fast properly for Day 0. How might this impact the results? Higher blood sugar on those initial glucose tests – which is exactly what was seen. The high glucose levels might reflect the unhealthy eating habits of the kids, as presumed, or it might reflect lack of compliance about the initial fast.
Statistical Analyses. Finally, the results of this study fail to account for multiple testing, and therefore may be overstating its results independent of the other issues mentioned. Multiple testing occurs when many statistical tests are run on data. When the data are independent, one might simply observe that, even if the data result from coin flipping, there is a .05 probability of finding data that are unexpected enough to “reject the null hypothesis”, i.e. to rouse suspicion that the data are not generated by coin flips. If a lot of different tests like this are made, the chance of finding something statistically significant is quite big, even if the data were obtained from coin flips.
In this case, the data are not all independent; glucose levels and insulin levels, for example, are correlated and need less correction for multiple testing than if they were independent. Yet these authors did no multiple test corrections—despite carrying out many statistical tests on their data. Were some of the results cherry picked among those tests that did not provide such strong evidence? Only a statistican could tell, and it seems the authors do not include one.
Another overlooked consideration relates to the variation among the kids. Were the results consistent over all groups of different ages, different puberty levels, and different sexes? Perhaps the study was designed to raise as many questions as it purports to answer.
The authors conclude that “Isocaloric fructose restriction improved surrogate metabolic parameters in children with obesity and metabolic syndrome irrespective of weight change.” But the claim that the diet contained the same calories as the usual sugary diet the kids normally eat is contradicted by their own data.
Thanks for this methodology analysis. I smelled a meth issue as I read through a media report on Lustig’s study, so googled to get the original paper; and found your treatment.
Diaries of food intake are certainly prone to reliability issues, no doubt more so for children and adolescents. Are we to believe that kids with disordered metabolism are suddenly to overcome significant chronic cravings, well, for 10 days?
And weight loss should have immediately excluded those subjects from the analysis.
I think what this study has highlighted though is a Calorie isn’t a Calorie, but not in the way Lustig self promotes. Rather, it is to do with variance in bioavailability of simple vs complex sugars. I am still vague on what on earth Lustig defines as “sugar”. it is not a term that should be used by scientists studying nutrition.
Lustig should account for the sloppy design. But this design also erodes faith in the journals that accept such loose methodology, especially in this field which the media jump all over to boost readership and profits.
Bruce Gray
Physiologist
Australia
To answer your question, if you watch Lustig’s talks, he specifies that it’s the fructose in table sugar (along with HFCS) that is toxic. Specifically, that it’s metabolized poorly only in the liver, and that creates issues that eventually result in weight gain and metabolic syndrome. He ends up comparing it to metabolism of alcohol in the system because supposedly fructose and ethanol create similar leftovers and toxins after being metabolized by the liver.
If his hypothesis is correct, you would actually expect to see weight loss in an isocaloric experiment where you severely restricted sugar. The weight in that case is a symptom, not a causal factor.
Also to the author, I’m sorry, but reducing your sugar consumption by 20% (okay 18% or so) of your daily calories a day is not a “small” reduction in sugar. It’s gargantuan. For a typical 2000 calorie a day diet, that would be going from around 600 calories a day down to 200. 400 calories of sugar is equal to eliminating 25 teaspoons of sugar a day out of your diet.
Even if we halve that for children at 1000 calories a day, you’re still going from 300 down to 100, or 200 calories, or around 12 teaspoons of sugar reduction per day. That’s still very significant. That’s not a “small sugar reduction”. Your choice of language taints the rest of the article by taking a hostile and incorrect tone from the very first sentence.
What I took away from the study is that it was done on a shoestring budget and did the best it could. But the results are interesting enough to warrant a better designed, funded, and controlled study.
You consider the entire article “tainted” by one aspect, but all the flaws in the study are acceptable, because budget? That seems inconsistent.
Say whatever you like about Lustig’s science, but having watched several of his lectures and explanations, I personally quit eating sugar, drinking fruit juice and cut back on nearly all fruit. And, I lost about 12 lbs in 2 months to a “normal” BMI for the first time in a decade. Also, my out of control triglycerides were cut in half, my HDL went up (that’s the good cholesterol), my LDL went down (that’s the bad cholesterol), and my total cholesterol is in the perfectly normal range. And my blood pressure dropped, too. None of that was due to exercise, for sure.
So, you all can poo-pooh this man’s work, and say his study is flawed, or that this another kind of “fad diet,” but I personally think the man is a genius. And I’m skeptical of all “diet” science. Cut the sugar and fructose folks — go with more whole veggies and less carbs & processed foods in your life.
Thanks for this methodology analysis. I smelled a meth issue as I read through a media report on Lustig’s study, so googled to get the original paper; and found your treatment.
Diaries of food intake are certainly prone to reliability issues, no doubt more so for children and adolescents. Are we to believe that kids with disordered metabolism are suddenly to overcome significant chronic cravings, well, for 10 days?
And weight loss should have immediately excluded those subjects from the analysis.
I think what this study has highlighted though is a Calorie isn’t a Calorie, but not in the way Lustig self promotes. Rather, it is to do with variance in bioavailability of simple vs complex sugars. I am still vague on what on earth Lustig defines as “sugar”. it is not a term that should be used by scientists studying nutrition.
Lustig should account for the sloppy design. But this design also erodes faith in the journals that accept such loose methodology, especially in this field which the media jump all over to boost readership and profits.
Bruce Gray
Physiologist
Australia
To answer your question, if you watch Lustig’s talks, he specifies that it’s the fructose in table sugar (along with HFCS) that is toxic. Specifically, that it’s metabolized poorly only in the liver, and that creates issues that eventually result in weight gain and metabolic syndrome. He ends up comparing it to metabolism of alcohol in the system because supposedly fructose and ethanol create similar leftovers and toxins after being metabolized by the liver.
If his hypothesis is correct, you would actually expect to see weight loss in an isocaloric experiment where you severely restricted sugar. The weight in that case is a symptom, not a causal factor.
Also to the author, I’m sorry, but reducing your sugar consumption by 20% (okay 18% or so) of your daily calories a day is not a “small” reduction in sugar. It’s gargantuan. For a typical 2000 calorie a day diet, that would be going from around 600 calories a day down to 200. 400 calories of sugar is equal to eliminating 25 teaspoons of sugar a day out of your diet.
Even if we halve that for children at 1000 calories a day, you’re still going from 300 down to 100, or 200 calories, or around 12 teaspoons of sugar reduction per day. That’s still very significant. That’s not a “small sugar reduction”. Your choice of language taints the rest of the article by taking a hostile and incorrect tone from the very first sentence.
What I took away from the study is that it was done on a shoestring budget and did the best it could. But the results are interesting enough to warrant a better designed, funded, and controlled study.
You consider the entire article “tainted” by one aspect, but all the flaws in the study are acceptable, because budget? That seems inconsistent.
Say whatever you like about Lustig’s science, but having watched several of his lectures and explanations, I personally quit eating sugar, drinking fruit juice and cut back on nearly all fruit. And, I lost about 12 lbs in 2 months to a “normal” BMI for the first time in a decade. Also, my out of control triglycerides were cut in half, my HDL went up (that’s the good cholesterol), my LDL went down (that’s the bad cholesterol), and my total cholesterol is in the perfectly normal range. And my blood pressure dropped, too. None of that was due to exercise, for sure.
So, you all can poo-pooh this man’s work, and say his study is flawed, or that this another kind of “fad diet,” but I personally think the man is a genius. And I’m skeptical of all “diet” science. Cut the sugar and fructose folks — go with more whole veggies and less carbs & processed foods in your life.
Excellent review. I think every obese child can be considered the control group. I also think that from a general knowledge stand point this study sheds light on what we already know.
I’ll admit, I’m not a scientist, but I am pretty certain that’s not how a control group works. Your last sentence sounds like you’re saying that since you already agreed with the study’s conclusions, you’re prepared to accept them based on truthiness. I don’t have a problem with science quantifying sugar’s effects on the body, but it should be precise in doing so. This study didn’t shed light on much aside from how shoddy methodology creates unreliable results. Hopefully someone tries a similar study, but in a more rigorous, responsible way.
Well put, non-science person. Awesome. Thank-you.
Excellent review. I think every obese child can be considered the control group. I also think that from a general knowledge stand point this study sheds light on what we already know.
I’ll admit, I’m not a scientist, but I am pretty certain that’s not how a control group works. Your last sentence sounds like you’re saying that since you already agreed with the study’s conclusions, you’re prepared to accept them based on truthiness. I don’t have a problem with science quantifying sugar’s effects on the body, but it should be precise in doing so. This study didn’t shed light on much aside from how shoddy methodology creates unreliable results. Hopefully someone tries a similar study, but in a more rigorous, responsible way.
Well put, non-science person. Awesome. Thank-you.
Goldin “is supported in part by an NSF grant” – so no vested interests then. . .
Goldin “is supported in part by an NSF grant” – so no vested interests then. . .