Originally posted June 18, 2013
Every now and then something big happens, so new and so different that you have to hold your breath to see if it realizes its full potential. Today is one of those days, thanks to two studies and four accompanying editorials in the Annals of Internal Medicine. First, the backstory: Medtronic developed a drug called recombinant human bone morphogenetic protein-2 (rhBMP-2), designed to promote bone growth after spine surgery, and made at least $1 billion from it. But this treatment wascontroversial, because several past studies had shown that it wasn’t any better than using a graft from the patient’s pelvis. Typically the drill when that happens is for researchers to go back and forth, with some studies showing benefit and some not, and a systematic review or meta-analysis trying to settle the question once and for all. But there are several problems with that approach.
First, the studies that get published are subject to a well-known “publication bias“—studies that show positive results are overwhelmingly the ones that make it into journals. Studies that show negative results may not even be submitted to journals, especially if they’re sponsored by the companies that make the product under scrutiny. But there are plenty of things researchers can and will do to make negative results into positive results without actually fudging the data. How? A few examples are reporting surrogate endpoints, or just the results for a subset of the population that showed the desired result, or making unrealistic comparisons (e.g. the new treatment vs. a placebo rather than the standard of care). Even when a well-meaning researcher performs a meta-analysis of both positive and negative studies, the data they’re using is still subject to those biases. The meta-analysis rarely eliminates bias; rather it amplifies it into something with more statistical power.
The only way around that is to get to the raw data itself, and that’s one of the amazing things about today’s studies. Through a collaborative process organized by the Yale Open Data Access (yes, YODA) project, two independent teams were selected to receive access to patient-level data held by Medtronic. (Note the distinction between patient-level data and already-published data: as I explained in the last paragraph, published data is subject to a variety of biases, whereas patient-level data suffers from fewer.) Today, they published two separate meta-analyses of that data in the same journal. Why two? In science, credibility goes hand-in-hand with reproducibility. If two teams, totally independently of each other, come to the same conclusions with the same data, it’s a lot harder to cast doubt on their analyses. And that’s more or less what happened today. Both meta-analyses independently concluded that rhBMP-2 doesn’t significantly reduce pain or improve recovery after spinal fusion. But there were notable differences in each paper. That’s important too, because it shows how meta-analyses still have a human component, as rigorous and quantitative as they might be. They can vary in their aims, the studies that they capture, and in many other variables.
So what does this all mean for research, and for those of us who aren’t about to get spine surgery? First, as I’ve tried to explain above, it is a seismic shift from how research is usually conducted. Even in academia, data is held under lock and key, because data means publications, and publications mean prestige and promotions and tenure. So the concept of an industry opening up its data to researchers is truly unique. It’s also seismic in the sense that it may have huge ripple effects (think tsunamis of glorious clinical evidence) but we just don’t know if that’ll happen yet. We need to see if other companies will follow suit. As this accompanying editorial attests, GlaxoSmithKline has also announced plans to open its clinical trial data up to researchers. The crucial question is, will these companies be rewarded for it? The results of today’s studies, in and of themselves, are not good for Medtronic. But will its losses from rhBMP-2 be compensated by its gains in credibility and good faith? Medtronic and GSK are led by much smarter people than myself, and I’m sure they had good reasons (both altruistic and otherwise) for opening themselves up to the risks of independent analysis. For this to be a sustainable model, short of a sweeping legislative mandate, every other biomedical company needs to see that this approach may benefit its bottom line. Only then will this seismic shift in research really shake things up.
Karan is a student at Robert Wood Johnson Medical School and Duke graduate who previously worked in strategic research for hospital executives.