Hundreds, possibly thousands of health studies may be wrong because researchers ignored human fallibility
You may have never heard of the Energy Balance Working Group, but this collection of 45 experts on nutrition, exercise, biochemistry, and other related disciplines have collectively thrown a “House-like” wrench into the research literature on everything from obesity to cancer and heart disease. Gregory House, the fictional and fantastically brilliant physician played by Hugh Laurie in the eponymous TV show frequently found his patients wanting in the court of self-reported truth: “I’ve found that when you want to know the truth about someone that someone is probably the last person you should ask.”
This is more or less what the Energy Balance Working Group have concluded in an “expert report” recently published in the International Journal of Obesity. If you want to know the truth about how much someone eats and exercises that someone is probably the last person you should ask.
In the technical language of the report “Energy Intake” (EI) and Physical Activity Energy Expenditure” (PAEE) are measures that are routinely misremembered when people are asked in health surveys to recall what they have eaten and how much they have exercised. For example, when self-reported estimates of energy intake are compared with the “gold standard” laboratory technique for estimate energy expenditure in humans (doubly labeled water), people systematically underestimate what they eat by hundreds of calories a day. When it comes to physical exercise, people tend to over-report their exertions.
And yet, even though these limitations are widely known, much research continues to accept the gospel of memory as a basis for scientific conclusions about health:
“Subjective methods of consumption are not only used for studies related to energy balance and body weight, but also to draw conclusions that inform clinical practice and dietary recommendations for conditions such as chronic kidney disease, cancers, heart disease, diabetes and Alzheimer’s disease. The lack of validity of some messages intended for clinical application is highly disconcerting. Inaccurate dietary data collection can result in spurious associations and effects that can directly impact patient care.”
How many findings are now presumptively invalid due to such self-reported data? We don’t know, says the University of Alabama’s David Allison, one of the lead authors of the expert report. “We can say that all conclusions based solely on self-reported energy intake or expenditure cannot be considered substantiated.”
The problem is that self-reporting is a cheap and convenient source of data for research, while more accurate alternatives are either expensive and challenging or, as yet, more promise than reality (see sidebar). But this shouldn’t, the report says, provide a rationale for continuing to fund studies based on inadequate measurement tools. Funding needs to be directed to solve this problem now, given the clinical and societal need for accurate measurement of energy intake and expenditure. As the authors note, “in this case, the adage ‘something is better than nothing’ must be changed to ‘something is worse than nothing.’”