While the term “evidence-based medicine” wasn’t coined until the 1990s, we have long strived to improve medical practice by generating evidence through trials. One could argue that recorded medical trials started as far back as the Book of Daniel, with the frequency and rigor of the trials significantly increasing through the renaissance period and into the modern era.1
Naturally, a key element of improving evidence-based medical practice is improving the evidence available. To this end, the gold standard has long been the Randomized Controlled Trial (RCT). RCTs excel at the rigor with which they can examine a specific situation under idealized conditions or homogeneous populations; however, they often fall short at establishing practices that are translatable to the general patient population or reflect real clinical practice.
One tragic example of this is in the use of statins. In clinical trials of statins for cholesterol management, adverse events occurred in less than 5% of the patients. This rate jumps up to as much as 20% in the general population.2 This discrepancy is likely caused by the exclusion of older subjects from trials and the disproportionate enrollment of female subjects.3
This lack of generalizability of RCTs has led to increasing calls for more trials using a pragmatic design across medical fields.3-5 This post will look at where pragmatic studies are congruent with developing a strong evidence base for medical practice. Feel free to share your own experience with trials, good or bad, by posting a comment below.
Explanatory vs Pragmatic Trials
The terms “explanatory” and “pragmatic” have been used to differentiate trials since the 1960s.6 Explanatory trials focus on evaluating the efficacy of an intervention in a very controlled and well-defined setting. The questions best answered by an explanatory trial are if and how under ideal conditions an intervention works. At its extreme, an explanatory trial will exclude all possible sources of known confounding factors to reach maximum homogeneity in patients and environment. This results in trials with a high degree of internal validity, that is, trials which to a great degree of certainty answer the question posed for the environment and the patient group described.5
By contrast, a pragmatic trial primarily tests the effectiveness of an intervention in routine clinical practice. The less controlled nature of a pragmatic trial means it is difficult, or even impossible, to be certain how the intervention worked. In this case, the question instead being answered is how well an intervention works in real life. At its extreme, a pragmatic trial includes all factors that are likely to occur in real clinical practice. A pragmatic trial is designed for maximum heterogeneity to maximize external validity, i.e., a pragmatic trial is maximizing the degree to which the results of the trial are applicable to the full spectrum of patients and environments.5
The use of these two terms may give the false impression that a strict dichotomy exists within trial design. In fact, there is a flowing continuum between explanatory and pragmatic trials. Discussing all of the possible study types is outside the scope of this post but Portela et al have completed a thorough overview on the subject.7 Instead, the remainder of this post will discuss two of the more robust types of pragmatic trial designs available.
Pragmatic Randomized Controlled Trials
The Pragmatic Randomized Controlled Trial is a twist on a rigid randomized controlled trial which allows for more flexibility in the treatment plan and far fewer exclusion criteria for the patient groups. The INFORM study provided a perfect case study of where a pragmatic study can fit within the spectrum of evidence generation.
The INFORM study sought to answer the question: is there a difference in mortality between patients who receive the oldest compared to the freshest red blood cell transfusions?8 The study was labeled a pragmatic randomized controlled trial, it incorporated two concurrent randomized cohorts, but importantly, it did not prescribe specific patient conditions beyond receiving red cells nor did it set specific red cell storage time ranges for the cohorts. While these two parameters reduced the level of control in the trial, they greatly increased the applicability of the results to clinical practice. Had a significant difference been found between the cohort receiving the freshest possible vs the oldest transfusions in the inventory, then serious consideration of adjusting transfusion practice would have occurred.
In the discussion of the secondary analysis of the INFORM study, the authors highlight that the most rigorous way of specifically comparing red cell transfusions stored longer than 35 days vs transfusions stored for 7 or fewer days would be to specifically transfuse such red cells to two randomly allocated cohorts. However, such a trial is not only logistically difficult to conduct but also ethically questionable as such practice would lead to a significant increase in outdated blood. Thus, a pragmatic design was the only way to generate a large sample size and sufficiently answer the question at hand.
Pragmatic Quasi-Experimental Studies
A Quasi-experimental study investigates an intervention by measuring the variable of interest before an intervention (control period) and after an intervention (intervention period) within the same study site. These studies can often remove some of the logistical and ethical hurdles of a randomized trial while still providing very valuable insights. The one potential drawback of this type of study is the risk of a temporal trend in the variable of interest.7
However, this limitation can be largely mitigated either by doing a detailed investigation and comparison of the patient conditions and medical practices of the control period and the intervention period or by implementing an off-on-off time-series design. One such study of platelet activation status based inventory management was conducted by Cedars-Sinai and presented at AABB this year.9
Pragmatic studies are applied to the whole patient population affected by the proposed change in practice. This means pragmatic studies can show the real-life implications of the intervention in question. The use of a comprehensive patient population mitigates the risks of excluding patients who react to an intervention in a fundamentally different way as was seen in the statin trials. In the case of studying platelet transfusion outcomes, using a comprehensive patient population ensures that the rare patients who have an increased risk of becoming refractory are not systematically excluded.
Both rigid RCT and pragmatic studies have an important role to play in generating the data needed to inform evidence-based medical practice. RCTs are required to ascertain the specific mechanisms by which a new intervention works, and can provide clear insight into an intervention’s efficacy under ideal conditions, whereas pragmatic trials are required in order to be confident that an intervention will be broadly applicable in real practice. As the INFORM study showed us, immense value and understanding can be gained despite not setting up the most robust and strict trial possible. When designing studies, it is important to remember that this paradigm is not a strict dichotomy, but rather a continuous spectrum of varying levels of internal and external validity. Understanding the range of possibilities and taking both types of studies into account will help us make better-informed decisions.
- Claridge JA, Fabian TC. History and development of evidence-based medicine. World J Surg 2005;29:547-53.
- Fernandez G, Spatz ES, Jablecki C, Phillips PS. Statin myopathy: a common dilemma not reflected in clinical trials. Cleve Clin J Med 2011;78:393-403.
- Maningat P, Breslow JL. Needed: pragmatic clinical trials for statin-intolerant patients. N Engl J Med 2011;365:2250-1.
- Axelrod DA, Hayward R. Nonrandomized Interventional Study Designs (Quasi-Experimental Designs). In: D.F. P, J.T. W, eds. Clinical Research Methods for Surgeons: Humana Press; 2006.
- Patsopoulos NA. A pragmatic view on pragmatic trials. Dialogues Clin Neurosci 2011;13:217-24.
- Schwartz D, Lellouch J. Explanatory and pragmatic attitudes in therapeutical trials. J Chronic Dis 1967;20:637-48.
- Portela MC, Pronovost PJ, Woodcock T, Carter P, Dixon-Woods M. How to study improvement interventions: a brief overview of possible study types. BMJ Qual Saf 2015;24:325-36.
- Heddle NM, Cook RJ, Arnold DM, et al. Effect of Short-Term vs. Long-Term Blood Storage on Mortality after Transfusion. N Engl J Med 2016;375:1937-45.
- A Supplement to TRANSFUSION Abstract Presentations from the AABB Annual Meeting Boston, MA, October 13-16, 2018. Transfusion 2018;58 Suppl 2:6A-254A.