When and How Can Real World Data Analyses Substitute for Randomized Controlled Trials?

    loading  Checking for direct PDF access through Ovid


Randomized controlled trials (RCTs) are generally considered by regulators to be the gold standard for establishing the causal relationship between medications and patient outcomes. However, RCTs are often costly, take a long time to complete, and are applicable to only a very narrow patient population. For rare diseases, recruiting a sufficient number of patients into a traditional randomized trial may be impossible. For lethal conditions with no available treatments, randomizing patients to receive highly promising treatment may be unethical. For these reasons, some have begun to suggest alternatives to RCTs. The 21st Century Cures Act and the proposed renewal of the Prescription Drug User Fee Act encourage the use of “real world evidence” (RWE), defined as data “derived from sources other than randomized clinical trials,” for regulatory decision making.1 This definition can include nonexperimental data primarily collected for research purposes, such as surveys, large cohort studies, and registries, as well as transactional real world data (RWD) created by the routine operation of the US healthcare system, such as health insurance claims or electronic health records (EHRs), among others (Figure1). Focusing on the definition of RWE given above, we do not discuss pragmatic randomized trials in this paper, although they are often considered RWE.
Longitudinal healthcare databases, such as claims and EHRs, are by far the most frequently used data source for RWE seeking to understand the use of medications and their safety and effectiveness in clinical care.3 Transactional databases provide detailed longitudinal records of the care and clinical outcomes of millions of patients, and they continue to grow in size, clinical detail, and accessibility through data linkage, standardization, and sharing. Studies based on longitudinal healthcare databases can evaluate drug effects in populations often excluded from RCTs, such as pregnant women, older adults, and patients with many comorbidities, and they reflect community‐based care and comedication patterns. They can also be completed relatively fast and at a small fraction of the cost of RCTs. Regulatory agencies have been using database analyses for decades to assess the safety of medical products, and they consider them for decision making for drug approval in select cases.1
Despite such uses, there remain major concerns about RWD analyses.8 Decision makers prefer RCTs not only because of their baseline randomization, but also because RCTs have the ability to tightly control measurements of patient characteristics and health outcomes and because their principles are easy to communicate. Properly designed and executed RCTs provide a priori confidence that their findings will be causally interpretable and useful for regulatory decision making. In contrast, database studies lack randomization and primary data collection, and they are often perceived as complex, poorly reported, inscrutable, and, therefore, harder to interpret and reproduce (Figure2). Although there is great potential for utilizing RWD analyses to supplement and sometimes substitute for RCT evidence on marketed medications, this lack of confidence in nonrandomized RWD analyses has limited their impact.
When RWD analyses and RCTs have been compared, many investigators conclude that, on average, both approaches result in similar findings.9 However, in individual settings, there can be substantial differences between RWE and RCTs. Although RWD analyses often ask slightly different clinical questions than RCTs and not all RCTs can be considered gold standard analyses,16 this finding is concerning. In some of these situations, findings have differed not only in the magnitude of estimated effect, but in direction, resulting in qualitatively different causal conclusions.18 These instances are often caused by perfectly avoidable mistakes in study design or analysis,20 and there have also been several examples of successful RWD analyses of therapeutics with findings later confirmed by RCTs.
    loading  Loading Related Articles