Almost half of all the clinical trials ever conducted have never been reported.

This is the story of the campaign to find them—and to fix medicine.

“I felt like I was a mouthpiece for this giant army of disenfranchised nerds,” said Ben Goldacre. It is a bright spring day in Oxford—a city of concentrated architectural stimulation—and we, along with Síle Lane, campaigns director for Sense About Science, are ensconced in a bland office at the very studious Center for Evidence Based Medicine, where Goldacre is a Senior Clinical Research Fellow.

The story of AllTrials really begins with blogging, which Goldacre started to do as a young doctor in 2003. “Bad Science” was an attempt to explain how science and statistics really work and how “quacks, drug companies, politicians, and dodgy journalists” repeatedly get science and statistics wrong, he said. And it hit a nerve: “On the one hand, mainstream media produced stuff that was clearly wrong, and on the other hand, mainstream media dumbed down and didn’t produce stuff that was stimulating and interesting and correct,” he says. His blog gained a following, and caught the interest of the Guardian newspaper, which gave him a column.

Inevitably, the ideas driving the blog and the column produced a book—a bestseller—Bad Science in 2009, which has sold over half a million copies in 25 countries, and then a follow up, Bad Pharma in 2012. “The central thrust of Bad Pharma is simple: trial results are withheld, and this breaks our efforts to practice evidence based medicine. I get bored by anecdotes about individual treatments. I’m much more interested in the structural issues, the principles and methods of science than I am in the actual answers,” he said. “So I don’t care which treatment works best. But I am unhealthily, weirdly obsessed in how you find out what works best—how you make sure you’re using fair tests that give you the most accurate possible answers. That’s why I think withheld trial results is a really interesting issue; it’s not about one treatment, it’s about this huge, festering hole that prevents us finding out what works best throughout all of medicine.”

This hole was not small. There is a vast amount of information on treatment—estimated at 50 percent of all the clinical trials ever conducted—that has simply gone missing in action. Who knew what insights—good or bad—lay in that data? Initial reaction to Goldacre’s book—from the pharma industry in the UK and even from government sources—was that this problem had been fixed, even though it wasn’t. “I was genuinely surprised to see the UK pharma industry body dive straight in with outright denialism: ‘These stories are all from the past; these problems have been fixed; trial results are posted on clinicaltrials.gov.’” But under pressure from Sarah Wollaston a doctor and a member of parliament, the government’s Minister of Health at the time, Earl Howe, eventually conceded that there was indeed a problem, and that he was grateful for the hard work Dr. Goldacre in attempting to raise the issue.

Nothing stops change quite like gratitude: it gives the news media nothing to storm at; it immobilizes the lever of shame. Yet here was “something that affects the whole of medicine and indeed the whole of science,” said Goldacre. “People have known about this problem for three decades, and they haven’t fixed it.”

Sile Lane: And that’s why Sense About Science’s ear pricked up. It’s something that we could change, and a change that would make a huge impact.

Ben Goldacre: It’s a fixable flaw.

Síle Lane: Ben had very clearly thought really, really, hard about the issue, and he set out the state of play: who’s talking about this, who’s involved in this from the academic side and from the policy side, and what the legislative levers might be. He told us about the European Union Clinical Trials Regulation update that was about to go through the legislative process in Europe, and he had all the facts and figures that would form some of the main messages of the campaign. The fact that only one in five trials on clinicaltrials.gov had actually published results—

Ben Goldacre: —despite the FDA Amendments Act.

Síle Lane: Only one in five that should have complied with US law actually did—and no one is being fined for that. All of these things that made it clear to us that something could be done here. There was a clear articulated set of problems with lots of evidence behind them. There were policy levers that could possibly be moved. And there were lots of people around the world talking about it. We had the ingredients to run a campaign. What we needed to do was to figure out what the outcome should look like and how we were going to get there.

Ben Goldacre: All trials registered, all results reported.

Síle Lane: AllTrials

Ben Goldacre: It was a great catch phrase, and it came from Tracey Brown (Director of Sense About Science in the UK).

Síle Lane: Another thing Tracey pointed out was that change would only come through public pressure on this issue. People needed to sign up to a positive statement saying we want this to change. And the people whose voices could most powerfully say that were patients.

 Bringing in the patients

Síle Lane: One of the things we do at Sense About Science is to bring different voices together to talk about how problems can be solved. It immediately occurred to Tracey that there had to be thousands of people who took part in clinical trials who felt “I want my participation to count for something—I want the results that you who ran the trial got because I took part to be used.” People in our office started thinking: “I know someone who was in a clinical trial! I’m going to call them now and talk to them about whether they care about this.” And they did really care about it! They told us why in their own words. Then we put their voices to work.

We started with an angry, open letter to the European Medicines Agency (EMA) from 50 people who’d been in clinical trials. The letter said:

“We all agreed to participate in the trials in the belief that we were helping to improve knowledge and treatments. We now understand that many participants in trials have been misled. Current evidence shows that, overall, about half of all clinical trials have not been published and that this proportion has seen only a small improvement over the past few years. … This is dangerous and expensive and it holds back good medicine. It is also a betrayal of our trust in clinical trial regulation, and the trust of the families of those patients who volunteer for trials having had a terminal diagnosis.”

The national papers in the UK picked it up. One of the patients who signed the letter, Richard Stevens, appeared on the One Show, the prime time chat show on BBC TV, and then another was interviewed by Channel 4 News. This was the real start of the campaign, when patients stood up in public and said, “We want you to share our data, so get on with it.”

Simultaneously, we had set up a rough website (built by volunteer web designers who cared about the issue) with a petition. The wording for the petition came from the early conversations at the SAS office. It said:

“All trials past and present should be registered, and the full methods and the results reported. We call on governments, regulators and research bodies to implement measures to achieve this.”

Within two weeks, we had 10 thousand names on the petition. Within the first month, 20 thousand people had signed up

Ben Goldacre: The petition thoroughly kicked the legs out from underneath anybody who was trying to pretend that this problem doesn’t exist. We made it, absolutely, impossible for people to say, “This problem has been fixed,” and we made it, absolutely, impossible for people to pretend that this was a fringe concern.

Síle Lane: And it wasn’t just patients or members of the public who were joining up. It was organizations with roles to play in clinical trial reporting. We spent a lot of time before we launched talking to professional societies and research organizations, such as the Wellcome Trust and the Medical Research Council. And when they signed on, it became pretty clear that the campaign was becoming a focus for those who wanted to be part of making the change.

Ben Goldacre: A big anxiety that people had with trial transparency is, “what if you find that I’ve got a trial that I should have posted 12 years ago and I didn’t.” Our response was that, rightly or wrongly, that was the cultural norm of the time. But we’re changing this and anybody who does the right thing deserves to be applauded. Anybody who even stands up and says “I’m going to fight for other people to join me in trying to do the right thing” deserves to be applauded.

Síle Lane: those were the conversations we were having with the first organizations who joined up.

Ben Goldacre: And that is the kind of thing only Sense About Science could have done. Firstly, there’s no way in a million years that one person could have done all this for fun, or that any group of people working collectively could’ve done it in their spare time. Lobbying effectively to create policy change and culture change is a job. It’s not the icing on the cake, it’s not the cherry on the icing, it’s the meat and drink, it’s the core activity. That is, obviously, what Sense About Science brought. They brought knowledge of how to do that and also the ability to recruit and supervise people who would do that.

Síle Lane: But it wasn’t just SAS doing it of course. We had got together before we launched with a group of the academics and publishers who were already immersed in discussions about clinical trial reporting—this became the steering group of the campaign. These people have more knowledge and contacts than you can even imagine.

The game changer: GSK supports AllTrials

Síle Lane: there was, of course, one unexpected development which changed the face of the campaign: GSK joined. Suddenly, all the arguments from other companies who were saying “This change isn’t needed and it will never happen anyway”were just completely swept off the table.” We were able to say, “Look, one of the biggest among you, has said they want to be part of this.”

Ben Goldacre: At the same time, there were lots of people saying, “GSK has withheld data and harmed patients over the years.” We said, “Yes. Like all companies that’s exactly correct. Like academic trialists have too.”

It’s really interesting the way that people easily slip into talking about transparency as if it’s an industry problem when actually non-industry trials are just as bad in failing to report results. It’s a universal problem. I think the reason why everybody focuses on industry is actually very simple. When you go to academics and say, “all trials should be reported, and it’s a scandal that they’re not, and it impacts negatively on patient care,” they go, “yeah, I know, that’s probably right.”

But when you go to industry they go on the attack: “you’re a charlatan who wants to put people’s full medical records on the Internet! You would destroy innovation overnight! This problem doesn’t exist anyway, and why are you being so shrill? And I’m going to sue you, and there’s already a law on this, and you’re very stupid for not knowing that this law exists, and you see regulation 43 stroke B stroke 17, and here’s a 20-page thing for…” They behave like children.

They pretend the problem doesn’t exist and they smear you, and that’s why they get the attention. That’s why they’re the focus of all the heat. And then they kind of flip between this and schmoozing PR: “we recognize that transparency is important to our stakeholders and we’re working collaboratively with all of our partners across the industry to ensure that we get adequate forward progress on this important issue.”

Once GSK signed up, that made it impossible for people to pretend that this problem doesn’t exist, and that is the single most important thing they could ever have done.

So yes, some bloggers, commentators, and doctors in academia reacted negatively by saying “You shouldn’t have anything to do with these people.” And I would strongly agree: GSK have done things I find appalling. But I also believe in people changing their minds, if only on certain issues, or shifting their priorities; and I think people need to work together to fix big problems. GSK signing up bridged an enormous empathy gap, it acted as a shorthand for “this is not a fringe concern,” and they kicked the legs out from underneath crass industry denialism. I’m not sure anybody on either side quite understood how big that was.

Síle Lane: When they joined the campaign, as well as saying “we think something needs to change” they announced the practical steps they had decided to take to make some of that change. They announced they were going to start making results available back to the year they formed as a company—2000—and they were willing to talk about things like the resources that was going to take, the timelines they were working to, how much time it takes to prepare a Clinical Study Report for publication—the nuts and bolts of transparency. That’s what I mean by making the campaign real.

Europe and the 300 lobbyists

Síle Lane: The next milestone for the campaign was working with thousands of our supporters on the European clinical trials law.

Ben Goldacre: That was really legwork intensive.

Síle Lane: The clinical trials regulation from 2002 was being overhauled, and while there were a few things in the new draft law about clinical trial transparency the legislation was full of loopholes. Our thought was, “We’ve got to make this right because this is going to be the law on clinical trial reporting in Europe for the next decade or more.” It was just after GSK joined the campaign that Glenis Willmott, a British Member of the European Parliament called our office and said, “I’m working on this, join me. ”

Ben Goldacre: We’ve been very lucky. Also, weirdly, Glenis’s righthand man was doing a part-time Masters in public health in the University where I was teaching!

Síle Lane: Well, you make your own luck. If we weren’t doing this, she wouldn’t have come to us. European policy making was a bit of a black box to us, and Glenis and her team helped us understand the state of play and what was up for grabs. But we had, at that point, 20,000 people signed up to the AllTrials petition, organizations such as the European Public Health Association, and BEUC, the European consumer group. So we had a European support base that we could mobilize when the time was right.

We also learned that the pharma industry had more than 300 lobbyists in the corridors of power in Brussels, 300 of them! Lobbying against transparency and for the status quo! When we heard that we kind of thought, “well—that’s that then.” There aren’t 300 of us, we’re not in Brussels, and we’re not lobbyists.”

But right turned out to be might. Thousands of campaign supporters contacted their MEPs to tell the MEPs in their own words why they cared about clinical trial transparency. There were letters from doctors, from patients, from people whose mom had diabetes saying, “I care about this.” And we eventually heard from five or six of those MEPs that those letters made a difference. Those letters helped them overturn the arguments they were hearing from the pharma lobbyists. 

As the law went through all the various procedural stages, we and our supporters kept on the pressure. It was a huge amount of work, but what we did was give structure and direction to the people and groups who supported the petition. When it came to the very last vote, it was almost unanimous in favour of more transparency. The result is law that is not perfect, but it’s pretty good.

Ben Goldacre: It’s really important to note that many of the campaign’s successes happened because there are good people in very senior positions who have to publicly appear to have never spoken to you, but who meet you in a hotel bar for an hour to make sure that they’re asking for the right things. Equally there are people who are often in quite low to middle positions within organizations, who go, “this is above my pay grade but I think if I ask for it, or I can push it I can make sure it goes higher up.” There is always a person who is inside that organization and who triggers action on this stuff, and nobody will ever know the efforts that they went to keep this all going.

Síle Lane: What the campaign did, really, was give people who are already talking about this and doing something towards it the support—the courage even—to keep going.

Ben Goldacre: We gave them evidence and strategic focus. We’d say: “this is exactly what your organization could do better at this time to make this happen. This is the opportunity that’s happening over there that you could get your organization to harness.”

Síle Lane: That’s what a campaign is.

The World Health Organization

Síle Lane: The World Health Organization was another milestone for the campaign. For over a decade, the WHO has said that all clinical trials should be registered. In 2014 it decided it needed a position on clinical trial results disclosure for the first time.

Ben Goldacre: I think it’s worth saying, someone good in the WHO decided one day that this is an important and useful thing that the WHO could take a stand on to benefit patients, and they began working towards that in their office.

Síle Lane: The WHO published a draft statement last winter. The statement was a good start, it said the results from trials should be published within a year of the trials end; however, it didn’t say anything about results from past trials, which was a shame, because most of the medicines we use today were tested in trials decades into the past. We, of course, welcomed the WHO’s statement and applauded it; but then we marshaled our campaign supporters to write to the WHO and say, “good stuff, but we think you could make your position stronger in these particular ways.”

Several months later, in April 2015, they published their final position. It was fantastic. It called for all clinical trials, past and present, to be registered and results from them reported within a year of their end, which is exactly what we had been arguing should happen.

At the same time the WHO set out their thinking behind their new position in a piece for the journal, PLOS Medicine, and they called for all regulators, companies, academics, institutional research boards, universities, funders—everybody who has a role in this—to take up the challenge in their own jurisdictions and make it happen.

And that’s where Ben comes in and says, “How you make this happen is auditing and benchmarking,” which is exactly what the next step of the campaign is going to do.

“You simply shouldn’t be allowed to run any more trials
until you make the results of your previous trials available.”

Ben Goldacre: An audit is the simplest tool we have in medicine. As a junior doctor, this is rammed down your throat. You’re forced to do an audit every other year, on waiting times, infection rates, and you have to pass multiple-choice questions on them; but they’re useful and really simple: you define the standard, you audit whether you’re meeting the standard, and then you review. You look at who’s doing well and who’s doing badly, you look at time trends and then you try and improve, and then you audit again.

What we need is rolling, routinely updated audit on every trial, where we go: “here are the trials that have been done, here are the trials that have reported results, here are the trials that have not.” Now when you do that you create several important outputs. Firstly, you can name and shame on individual trials; but if you audit in a systematic way across all trials on all registries you can also say, “the following companies are the best and the worst for withholding data.” You can say “the following drugs are the best and the worst for withheld data. The following professors are the best and the worst for withheld data. The following institutions are the best and the worst for withheld data.”

This is information of practical value. If people are performing badly, and if we assume good faith, then they can learn from the people who are performing properly, and sharing all trial results. This sort of thing should be baked into the funding of research, and regulators. It’s such an obvious metric.

After audit, the second thing we should spend money on—we, meaning, the world of medicine—is simple awareness raising: everybody involved in doing a clinical trial needs to make sure it gets registered. And at the end, we need people to know that anybody can post their results for free on clinicaltrials.gov. There there are no limitations to that, so there are no excuses. It doesn’t matter if you did your trial in Belgium or Azerbaijan, or Papua New Guinea, or Chicago, or New York or London; you can put the results on clinicaltrials.gov; and you should. Also, posting results there doesn’t stop you also being able to publish the results later in a journal, to cash in and make your resume look good as a scientist. Again, people need to know that. These aren’t trivial things. Communicating them universally would be easy, but it would have huge impact. It’s nuts, after spending millions on every trial, to fail to spend a tiny bit of money getting people to post all the results.

But that’s just the start. Ethics committees should be saying to everybody who applies to do a clinical trial, “are you sitting on the results of any clinical trials that have finished more than 12 months ago?” And if the answer is “yes,” then you shouldn’t be allowed to touch any more patients. You simply shouldn’t be allowed to run any more trials until you make the results of your previous trials available. You have betrayed the trust that those patients put in you as a researcher. Patients don’t participate in clinical trials for the commercial benefit of the company, or for resume points, or for the edification of the researcher; patients participate in clinical trials in order to improve the evidence base for patients like them in the future. If you don’t post the results, you’ve abused them.

We should also be providing performance metrics for government funders. And they should be using them—should be saying “we’re going to withhold 20 percent of your funding until you publish. We’re going to not give you another grant if you don’t publish on the last one.”

And doctors, of course, might want to think twice about using a treatment from a company with a poor track record on transparency, especially if there’s another treatment that seems to be equally good from a company you can trust more. This is information that can be used directly to improve patient care in an immediate and direct way. And it’s inexpensive to do.

The policy audit

Ben Goldacre: That’s all about performance. It will cost. But policy is easier to measure. Again, it’s important to reward good practice and it’s important to reward people who are forward thinking and trying to do the right thing. So what we’re doing is a policy audit. We set out a gold standard for transparency that could be applied to the pharmaceutical industry. We looked at whether a company registers all its trials, how far back in time it shares results, whether it audits its own compliance internally or externally, whether they commit to post results, and so on. And then we applied a scoring system to rank the results.

This was a really important exercise because. As we saw earlier, GSK made it difficult for people to say, “This is impossible.”

And what we found was that everything we say pharma should do—everything we ask pharma to do—has been done by at least one company. Nobody’s perfect, but everything we said people should do has been done by at least one organization and that means—

Síle Lane: —everybody can do all of this. The point of our audit is not just to show the good from the bad and to name and shame, but to produce a clear picture of what the best practices are, how they were implemented, and by which company. We will have all the details that can then be picked up and used by anybody else. And AllTrials is not just aimed at changing the pharmaceutical industry; we will try to make sure this knowledge is applied to the entire ecosystem of research funding—academia, government, non-industry funders.

Ben Goldacre: Which will be fun.

Síle Lane: At some point, we’ll have the whole ecosystem audited, and a clear set of best practices that anybody and everybody can absorb and use in their own sectors, as the World Health Organization called for.

The public gets it

Ben Goldacre: There’s a drinking game I did over one very long night with some doctors, on what medical practice of today will be looked back on, by future generations, with appalled astonishment—the way that we look back on medieval bloodletting. I really do believe that it will be this whole problem; the way we have let the results of so many clinical trials be so casually buried. It harms patients, it makes no sense, but more than that, it so obviously makes no sense. Running a clinical trial is an expensive business. It’s 10 thousand dollars a patient, and we spend that money specifically because we want to exclude bias from our findings and get the most accurate possible answer.

That’s the key. You don’t do a trial on whether a parachute will save your life jumping out of an airplane. And you don’t need to do a trial on treatments that really obviously work. We do trials, we spend all that money, because we’re trying to fine tune medical practice; we’re trying to detect quite modest shifts in the risks and benefits of new treatments versus old. These are two things that are really surprising to many people outside medicine: firstly, most new treatments are only slightly better than what we had before and, secondly, that’s still really useful, across a whole population.

But because we’re detecting small differences between one treatment and another, we need to be perfectly hygienic with our data. That’s why we do randomized trials, fair tests that are free from bias; and we do those trials—we spend millions and millions of dollars in each trial—because we want something better than going “well, I’m a doctor and I tried it, and my patients seemed to get better.” So we spend all this money to exclude bias; and then after spending millions, hundreds of millions, billions of dollars every year on doing randomized trials specifically to exclude bias… we allow all that bias and error to flood right back in because we allow a third, a half, of all trials to be routinely and illegally withheld from doctors, researchers, and patients.

It makes no sense at all. It is insane. I travel around the around the country giving lectures to the general public on this and they get it immediately. If I can explain this to them in under five minutes then why is this problem still going on?

Which side of history do you want to be on?

Ben Goldacre: This is an issue that’s been successfully dodged. Partly because it’s so absurd, that when you try to write or talk or lobby about it, people think you must be crazy and that this problem can’t exist. But at the same time, it’s also been protected from scrutiny by the public and policy makers because it’s a little bit dorky; it takes a little bit of reading before you get to the madness of it. So it’s been dodged. But the one thing that I think is important to communicate to the enemies of transparency is that we’re not going away, and we’re not alone. The one thing I think we’ve achieved is that we’ve made it impossible for people to dodge this issue.

Resistance is unwise, I think. There is a huge, inevitable policy trajectory towards greater transparency throughout the whole of society and in clinical medicine. It’s going to happen with clinical trials, and it’s going to happen retrospectively, including the results of the past. There is no doubt. It’s no longer sustainable to resist. The people who do it fastest and best will take a prize; and those leaders will be truly improving patient care, whether they’re from industry or academia.