When and How Can Real World Data Analyses Substitute for Randomized Controlled Trials?
Longitudinal healthcare databases, such as claims and EHRs, are by far the most frequently used data source for RWE seeking to understand the use of medications and their safety and effectiveness in clinical care.3 Transactional databases provide detailed longitudinal records of the care and clinical outcomes of millions of patients, and they continue to grow in size, clinical detail, and accessibility through data linkage, standardization, and sharing. Studies based on longitudinal healthcare databases can evaluate drug effects in populations often excluded from RCTs, such as pregnant women, older adults, and patients with many comorbidities, and they reflect community‐based care and comedication patterns. They can also be completed relatively fast and at a small fraction of the cost of RCTs. Regulatory agencies have been using database analyses for decades to assess the safety of medical products, and they consider them for decision making for drug approval in select cases.1
Despite such uses, there remain major concerns about RWD analyses.8 Decision makers prefer RCTs not only because of their baseline randomization, but also because RCTs have the ability to tightly control measurements of patient characteristics and health outcomes and because their principles are easy to communicate. Properly designed and executed RCTs provide a priori confidence that their findings will be causally interpretable and useful for regulatory decision making. In contrast, database studies lack randomization and primary data collection, and they are often perceived as complex, poorly reported, inscrutable, and, therefore, harder to interpret and reproduce (Figure2). Although there is great potential for utilizing RWD analyses to supplement and sometimes substitute for RCT evidence on marketed medications, this lack of confidence in nonrandomized RWD analyses has limited their impact.
When RWD analyses and RCTs have been compared, many investigators conclude that, on average, both approaches result in similar findings.9 However, in individual settings, there can be substantial differences between RWE and RCTs. Although RWD analyses often ask slightly different clinical questions than RCTs and not all RCTs can be considered gold standard analyses,16 this finding is concerning. In some of these situations, findings have differed not only in the magnitude of estimated effect, but in direction, resulting in qualitatively different causal conclusions.18 These instances are often caused by perfectly avoidable mistakes in study design or analysis,20 and there have also been several examples of successful RWD analyses of therapeutics with findings later confirmed by RCTs.