The Audacity of Dope

The Audacity of Dope

by Jul 6, 2015Accuracy and precision, Studies in the News6 comments

“If you buy medical marijuana edibles, you are probably getting ripped off,” headlined Quartz. “Walk down any convenience store drug aisle and you can generally rely on the products on the shelves to contain the dosages they promise. But walk into a marijuana dispensary and you might not feel the same level of assurance,” said the Washington Post. “[T]his is an issue of poor accuracy, rather than pure ripoff strategies,” reported Gizmodo.”

The news followed on the findings of a study published in the Journal of the American Medical Association (JAMA) in which researchers examined THC levels in edible marijuana sold in three major cities on the West coast of the United States (San Francisco, Los Angeles, and Seattle).

In terms of THC content—a psychoactive component of marijuana —the authors found that only 17 percent of the products were accurately labeled compared to values of THC confirmed in a lab, while 23 percent contained less THC than the labels indicated—and 60 percent tested for more THC than the label indicated. Seventy-five products from 47 different brands were tested; Los Angeles was more likely to sell products with greater THC than labeled, while Seattle was more likely to sell products with less THC than indicated. The cut off for accurate labeling was 10 percent. If a lab value was more than 10 percent greater or less than the corresponding label value, the label was considered “incorrect” by the study’s standards.

But were these lab numbers reliable? The problem is that many journalists seem to have misunderstood a critical distinction between accuracy and precision. And without noting the difficulties of the original lab work, the result was that the news media inadvertently confirmed the lab numbers as being true even though those lab measurements are, in fact, quite uncertain.

In the scientific world, the accuracy of an individual measurement is how close the measurement is to a value generally believed to be the truth. One aim of the JAMA study was to figure out whether the labels were accurate. Imagine for a moment that there’s a big “bullseye” on the actual THC level of the product. The study’s authors wanted to know whether the label hits the bullseye or not. The best approximation the authors have to the actual THC level, however, is the lab value:

Dope1

The authors assumed that the lab value could be substituted for the true value. But the true value of the THC level, as represented by that big red dot, isn’t necessarily captured by the lab; indeed, there is strong evidence from the lab’s reported results that it missed the mark for the true value.

One indication for poor accuracy is that the lab findings have low precision. Precision, in contrast to accuracy, is a measurement of how close the lab results come to one another. The lab carried out two measurements of the THC level for each of the 75 products. In 37 of these products (just under half), the two measurements were not within 10 percent of each other. In other words, in half the cases the lab considered, the two measurements were not close enough to each other to meet the study’s own standard of precision for the label.

Of course, it is possible that the lab’s results are extremely accurate, and we’re observing variation in the products themselves; perhaps, the marijuana was not mixed uniformly into the pot biscuits. But we have no way to tell, as the study did not provide an explanation for the discrepancy: in these cases, the scientists simply homogenized the entirety of the product they purchased, and retested, obtaining a result that they used as the “true” result.

Pictorially, here’s what we might be looking at:

Dope2

Will the valued of the homogenized products tested in the lab be closer to the red dot than the original two measurements? Perhaps. Are the original lab-measured values of THC an accurate approximation of the true THC value for the 39 products whose two measurements were close to each other? Perhaps. Is the label’s value closer to the true value than the laboratory values? Perhaps.

So if the lab has low precision, can it have high accuracy? In general, labs can be highly precise, and have low accuracy (sometimes called “trueness”). Imagine, for example, a clock that goes too fast; if you use it to measure how long something takes, you will have precise measurements (all clumping together), which systematically overestimate the time.

It is also possible to obtain results that are accurate on average, but aren’t very precise. If for example, your clock is a little erratic, randomly overestimating or underestimating the time, then on average you might have a very good measurement, but the individual measurements are not that close to each other. They key is that one can only have an accurate average without precision if there are many measurements. Even then, you still need to presume there is no bias in the lack of precision.

In this case, the lab did not take many measurements—just two per product, and in some cases the measurement for the homogenized product. Without larger numbers, the lack of precision is a clear indicator that the overall results are inaccurate, and therefore unreliable.

We may want to have faith in the laboratory results, but its imprecision is a clear warning: why should we believe the true THC level is that indicated by imprecise lab results, rather than the label (which presumably derived from a different lab’s handiwork)? If individual pot products themselves have different amounts of THC in different “parts” of the food, then the study’s design should have indicated how they came to this conclusion, and each product should have been homogenized, rather than just those whose measurements varied significantly.

No doubt, this study calls into question the labeling on edible marijuana products, as several media sources reported. But the methods also call into question the process by which we make these measurements, and suggest that poor precision with too few measurements of the THC content in foods may plague our checks on label accuracy.

Rebecca Goldin is Professor of Mathematical Sciences at George Mason University and Director of STATS.org. She received her undergraduate degree from Harvard University and her Ph.D. from the Massachusetts Institute of Technology. She taught at the University of Maryland as a National Science Foundation postdoctoral fellow before joining George Mason in 2001. Her academic research is in symplectic geometry, group actions and related combinatorics. In 2007, she received the Ruth I. Michler Memorial Prize, presented by the Association for Women in Mathematics. Goldin is supported in part by NSF grant #1201458.
Main image adapted from © OSTILL
Please note that this is a forum for statisticians and mathematicians to critically evaluate the design and statistical methods used in studies. The subjects (products, procedures, treatments, etc.) of the studies being evaluated are neither endorsed nor rejected by Sense About Science USA. We encourage readers to use these articles as a starting point to discuss better study design and statistical analysis. While we strive for factual accuracy in these posts, they should not be considered journalistic works, but rather pieces of academic writing. 

6 Comments

  1. Joe Severs

    A few thoughts from my experience in the pharmaceutical world:
    1) “If a label value was more than 10 percent greater or less than the lab value, it was considered “incorrect” by the study’s standards”: typical pharma practice is to divide lab value by label value. Plus/minus 10% is typical.
    2) Comparing variation between 2 values (“two measurements were not within 10 percent of each other”) indicates inaccuracy (“the two measurements were not close enough to each other to meet the study’s own standard of accuracy”) is mixing up precision with accuracy. The best assay method may be imprecise for a variety of biochemical reasons, but that does not make it inaccurate.
    3) “If individual pot products themselves have different amounts of THC in different “parts” of the food . . .”: this is content uniformity.
    4) ” . . . poor precision in measuring THC content in foods may plague our checks on label accuracy.” If the assay is imprecise, then you take more samples, perhaps a lot more, to obtain a mean value with tolerable precision. Instead of ‘plague’ I would use ‘hamper’; the additional testing is the manufacturer’s cost, but it can be done.

    My thanks for the article. Joe

    Reply
    • statsorg

      Thanks for those insightful comments; we’ve made changes to the piece as a consequence.

      Reply
  2. Joe Severs

    A few thoughts from my experience in the pharmaceutical world:
    1) “If a label value was more than 10 percent greater or less than the lab value, it was considered “incorrect” by the study’s standards”: typical pharma practice is to divide lab value by label value. Plus/minus 10% is typical.
    2) Comparing variation between 2 values (“two measurements were not within 10 percent of each other”) indicates inaccuracy (“the two measurements were not close enough to each other to meet the study’s own standard of accuracy”) is mixing up precision with accuracy. The best assay method may be imprecise for a variety of biochemical reasons, but that does not make it inaccurate.
    3) “If individual pot products themselves have different amounts of THC in different “parts” of the food . . .”: this is content uniformity.
    4) ” . . . poor precision in measuring THC content in foods may plague our checks on label accuracy.” If the assay is imprecise, then you take more samples, perhaps a lot more, to obtain a mean value with tolerable precision. Instead of ‘plague’ I would use ‘hamper’; the additional testing is the manufacturer’s cost, but it can be done.

    My thanks for the article. Joe

    Reply
    • statsorg

      Thanks for those insightful comments; we’ve made changes to the piece as a consequence.

      Reply
  3. Gregory Gruska

    Thank you for identifying a an often overlook element of studies — that related to the measurement system variation. I am surprised that a journal such as JAMA does not require what is common in the non-medical industries; that is, the identification of the uncertainty of provided results. In many industries, laboratories must be certified to a national or international standard such as ISO 17025 which requires the laboratory to quantify the uncertainty of its measurement processes and use this information when reporting results (usually the 95% confidence bounds of the measurements.) Instead of picking an arbitrary percent to evaluate “correctness”, a statistical evaluation (e.g. hypothesis test) can be made. Further, the focus of the study should have been better defined. As you pointed out, the homogeneity of the products was a late consideration. This implies that the (implied) study was to see that every part of the product met the label even though the product preparation process for edibles is not know for this feature.
    Thanks for the article. We can only hope it will have a positive impact on future studies.

    Reply
  4. Gregory Gruska

    Thank you for identifying a an often overlook element of studies — that related to the measurement system variation. I am surprised that a journal such as JAMA does not require what is common in the non-medical industries; that is, the identification of the uncertainty of provided results. In many industries, laboratories must be certified to a national or international standard such as ISO 17025 which requires the laboratory to quantify the uncertainty of its measurement processes and use this information when reporting results (usually the 95% confidence bounds of the measurements.) Instead of picking an arbitrary percent to evaluate “correctness”, a statistical evaluation (e.g. hypothesis test) can be made. Further, the focus of the study should have been better defined. As you pointed out, the homogeneity of the products was a late consideration. This implies that the (implied) study was to see that every part of the product met the label even though the product preparation process for edibles is not know for this feature.
    Thanks for the article. We can only hope it will have a positive impact on future studies.

    Reply

Trackbacks/Pingbacks

  1. Maconha gera confusão | Embasado - […] Infelizmente, a maioria dos comentários falou bobagem por confundir termos fundamentais. Um artigo no site da American Statistical Association…
  2. Maconha gera confusão | Embasado - […] Infelizmente, a maioria dos comentários falou bobagem por confundir termos fundamentais. Um artigo no site da American Statistical Association…

Submit a Comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Share This