Scott Miller, PhD, is a biostatistician at Clinipace.
I had a great time hosting the webcast, Navigating Regulatory Biostatistical Requirements through the Trial Lifecycle, with my colleague Ron Marks, PhD, Clinipace CSO, director of biostatistics, and cofounder. We shared information on common statistical issues and potential solutions for the planning, conduct, analysis, and submission phases of a clinical trial.
Following the webinar, we started a series of blog posts to address some of the excellent questions from attendees. In the first group of Q&A posts, we answered questions regarding the trial planning phase, including protocol deviations, adaptive designs, and alpha spending functions in interim analyses.
Below we address questions related to trial conduct. If you have any additional thoughts or related questions to ask our experts, please share your comments below.
Q: If you perform an interim futility analysis, will it impact the Type I error if you don’t consider stopping for superiority?
A: Due to the fact that futility analyses and efficacy analyses interact in ways that impact each other, both type I and type II errors can be affected when a trial has both options. However, futility analyses can sometimes impact the trial’s overall Type I error even if you don’t assess for efficacy. The reason is that if you imagine a trial where the treatment doesn’t work, concluding that it does work at the end of the trial (i.e. p<0.05) would be a Type I error. We’d expect that to occur about 5% of the time due to random sampling variability.
To examine Type I and Type II errors, statisticians often explore what are called simulation studies, which just means that we use computers to simulate a given trial, run it some number of times, and then see how often various outcomes occur. If you think of a hurricane landing probability map, those are generated via simulations. So if we imagine our hypothetical trial and add in a futility analysis, some of those simulated trials will stop for futility at the interim analysis stage. If we hadn’t had a futility analysis option, some of those trials might have been deemed statistically significant, which would be a Type I error. But by adding in the futility analysis, some of those trials would be stopped early, and thus futility analyses can result in fewer Type I errors.
The FDA can become concerned about the impact of interim analyses on type I and type II error, and sometimes ask trial sponsors to specify whether a given interim analysis for futility and/or efficacy will be “binding.” A binding analysis means that if your futility analysis is significant, you have to stop the trial early, and similarly for early efficacy analyses. Sponsors often like to retain some flexibility on these actions in order to consider additional factors beyond the primary efficacy analysis, such as secondary efficacy results or safety profile.
Q: Can you discuss further key ‘operational biases’ to avoid?
A: “Operational bias” is a fairly broad term because it refers to any change to the conduct of an ongoing trial that could impact the conclusion of the trial. The most common source of these is making changes in response to something you observe in an interim analysis. A fairly egregious example would be performing an interim analysis and noticing that the results look more supportive in men than in women, then modifying the trial to enroll more men than women. A regulatory agency would see this change in the trial conduct as introducing operational bias because the change in enrollment population appears to have been made in order to increase the observed treatment effect.
Another example would be unplanned interim analyses. Maybe you plan to conduct one interim analysis once you’ve enrolled half of your target population. And when you see the results, you just miss statistical significance for stopping early for efficacy. But then you do another (non-pre-specified) interim analysis two months later. Regulators would be very concerned about that sort of thing, as it risks inflating your Type I error.
A final example would be if you conducted an interim analysis and your primary efficacy analysis was not particularly supportive but one of the secondary efficacy analyses was, then modifying the trial protocol and Statistical Analysis Plan (SAP) to make that secondary efficacy analysis your primary.
In general, operational bias is in the eye of the beholder. When that “eye” is a regulatory agency, it is extremely important to describe any interim analyses in the protocol and specifically address issues of who will see that type of data/conclusions. The safest approach is to have “firewalls,” such as only allowing unblinded results to be seen by personnel not otherwise involved in making trial conduct or analysis decisions. For example, a Data Monitoring Committee (DMC) could see the unblinded results, but their recommendations to the trial sponsor would not provide results. And often the statistician conducting an interim analysis is separate from the statistician who will be conducting the final analysis.
If you’re interested in learning more about Navigating Regulatory Biostatistical Requirements through the Trial Lifecycle, be sure to listen to our webcast in its entirety. Check back soon for posts about questions regarding trial analysis and submission as well as the related eBook!