Jasper van Egeraat, Bas Penning de Vries, Ewout Steyerberg
High risk devices may be approved based on limited evidence from underpowered clinical studies. They may have a higher risk than commonly accepted. While an observed event rate may seem favourable, regulators, healthcare professionals and patients should be aware that there is considerable uncertainty due to limited evidence.
To assess this uncertainty, we developed an online tool designed to provide users with objective insights into potential risks that may not have been detected during the approval process. The tool can assist making informed decisions regarding the use of these devices by demonstrating their implicitly accepted risk.
The tool uses Bayesian statistics with a constant hazard assumption to answer the following two questions:
- Based on current evidence, what is the upper bound to the n-year risk for an individual patient?
- If we collect more evidence, how does our confidence in the upper bound for an individual patients n-year risk change?
The tool requires users to input the collected evidence in the form of cumulative device experience, i.e. the total number of years the device has been implanted in humans, and the number of observed events during these years. The tool calculates an upper bound for the n-year risk for an individual patient, as a function of the cumulative device experience. Increasing the cumulative device experience, by collecting more evidence, will increase our confidence that the n-year risk is lower than a certain threshold. This can be seen in Figure 1, where we show the upper bound to the 10-year risk of a hypothetical device with only 100 cumulative patient years and the small number of 4 events. With this evidence, we are 90% certain the 10-year risk is below 55%. By collecting up to 800 cumulative patient years we can be 90% certain that the risk is below 40%.
We have successfully tested the tool using a dataset of studies comparing bioresorbable vascular scaffold (BVS) with an already approved Everolimus drug-eluting stent. Figure 2 shows that based on the results of the trial, the 10-year risk for an individual patient with the inferior BVS device is higher than for the drug-eluting stent.
Further methodological research may implement a non-constant hazard. Considering more use-cases with real-world data is also recommended.
The tool can be accessed via the following link:
https://core-md.shinyapps.io/RiskCalculator/.
Moreover, the code and the dataset used for the testing are openly available in the Zenodo CORE-MD community here: https://zenodo.org/records/11366097.
