What Does It Mean to Say a New Drug ‘Works’?Feedzy

John Beigel’s problem was that not enough people were dying of Covid-19. Not that he wanted them to, of course. It’s just that, as bad as the pandemic has gotten, it hasn’t killed as many people as it seemed like it might. And Beigel, a researcher at the National Institute of Allergy and Infectious Diseases, was designing a big study to see if an experimental antiviral drug called remdesivir would work against the disease. He and his team needed the right “end point”—the thing they could count, turn into data, and analyze.

Mortality is a great end point. It’s right there in the name.

“But when we were thinking of end points, we thought a mortality study would need 3,000 or 4,000 people,” says Beigel, associate director for clinical research in NIAID’s Division of Microbiology and Infectious Diseases. Those high numbers would give his team enough “events”—which is to say, deaths—to get a statistically significant measurement of the drug at work. And they didn’t have enough time to get that many people enrolled in their US study. “We thought it was important to get the study done, to get a clinically meaningful end point, without taking the time we would need to do a mortality study,” he says.

This was back in February, before anybody really knew anything about Covid-19. So Beigel’s team tried a different tack, one familiar to scientists and regulatory agencies like the Food and Drug Administration (which issued the Emergency Use Authorization under which people are studying the drug). Participants in the study would get scores, every day, calculated on what’s called an ordinal scale. Healthy and released from the hospital, you get a score of 1. Dead, you get an 8. The other numbers were for everything in between—like whether the person has to get admitted to the hospital or needs oxygen or has to go on a mechanical ventilator.

Then they found another problem. “When we first wrote the protocol, the end point chosen was ordinal score at day 15,” Beigel says. “That is something we’ve used for influenza studies before, so we knew the FDA would be OK with it, and is something that actually matters to the subject.” That’s a good end point: It’s not just a statistical entity. It’s long enough to show an effect in many diseases, and it has clear clinical relevance.

“In March we started hearing reports that the course of the disease might be much longer, and that there were people in the hospital for three weeks, up to four weeks,” Beigel says. “What happens if the recovery is much later than day 15? You might actually have a significant difference, yet you wouldn’t show it.”

So Beigel’s team changed their end point: ordinal score at day 28. They hadn’t seen their data yet when they made the switch. That would’ve been an ethical no-no, chasing statistical significance by juking their methods. But juking ahead of data? Kosher, but they knew it’d be controversial. “It raised suspicion for our study,” Beigel says. “If this was something that is well known, like influenza, and we switched in the middle of the study, that would be really suspicious. For Covid, we haven’t seen anything like this.”

In a study published in late May in The New England Journal of Medicine (and previewed during a press conference at the White House), the team concluded that patients taking remdesivir recovered in a median time of 11 days, while people given a placebo took a median of 15. It was enough to get remdesivir added to the US standard of care, pronto—the first drug identified as having a beneficial effect on the new disease.

Read More