Levels of Evidence for Adult and Pediatric Cancer Treatment Studies (PDQ®)–Health Professional Version

The double-blinded RCT is the gold standard of study design. To meet this designation, the study allocation must be blinded to the physician both before and after the randomization and treatment assignment take place. This design provides protection from allocation bias by the investigator and from bias in assessment of outcomes by both the investigator and the patient. Unfortunately, most clinical trials in oncology cannot be double-blinded after treatment allocation because procedures or toxic effects often vary substantially among study allocations in ways that are obvious to both the health care professional and the patient.

Meta-analyses of randomized studies offer a quantitative synthesis of previously conducted studies. The strength of evidence from a meta-analysis is based on the quality of the conduct of individual studies. Moreover, meta-analyses can magnify small systematic errors in individual studies. A study comparing the results of single, large, randomized trials with those of meta-analyses of smaller trials published earlier on the same topics showed only fair agreement (kappa statistic, 0.35). Outcomes of the large RCTs were not predicted accurately by the meta-analysis 35% of the time.[1,2] Meta-analyses performed by different investigators to address the same clinical issue can reach contradictory conclusions.[2] Therefore, meta-analyses of randomized studies are placed in the same or lower category of strength of evidence as randomized studies, not at a higher level.

This category includes trials in which treatment allocation was nonrandomized and prospective. The allocation would be known to the investigator before informed consent is obtained from the patient. An imbalance can occur in treatment allocation under such circumstances. Subset analyses within randomized trials often fall into this category of evidence.

Subset analyses of randomized studies are subject to errors inherent in multiplicity (i.e., statistically significant results are to be expected because of random variation of measured effects in multiple subsets). Therefore, subset analyses do not represent the same strength of evidence as the overall analysis of a randomized trial as designed unless explicit prospective hypotheses are made for the analyzed subset. Subset analyses should be placed in the next lower category of study design, nonrandomized controlled clinical trials.

These clinical experiences are the weakest form of study design, but they may be the only available or practical information in support of a therapeutic strategy. This is especially true in the case of rare diseases or when the evolution of the therapy predates the common use of randomized study designs in medical practice. Case series or observational designs may also provide the only practical design when treatments in study arms are radically different (e.g., amputation vs. limb-sparing surgery). Nevertheless, they always raise issues of patient selection and comparability with other populations. In terms of generalizability to other populations, the strongest to weakest study designs are the following: population-based studies that have a definable population, nonpopulation-based but consecutive series, and nonconsecutive cases. Some study designs (e.g., cohort and case-control studies) have internal-control study subjects, while others do not (e.g., case-only series with no internal comparison group or case-only series that are compared with historical controls).

Even large, population-based, observational studies with internal controls that compare therapeutic strategies in oncology should be interpreted with extreme caution. In a study that directly compared observational studies with RCT results, investigators performed a systematic MEDLINE search for observational studies published from 2000 to 2016 using data from the Surveillance, Epidemiology, and End Results (SEER) Program, SEER-Medicare, or the National Cancer Database that compared treatment regimens for any diagnosis of cancer.[3] The investigators matched 350 treatment comparisons to 121 RCTs that made the same comparison. They found no significant correlation between the hazard ratios (HRs) of the observational studies and the matching RCTs (concordance correlation coefficient, 0.08; 95% confidence interval [CI], -0.07 to 0.23). Only 40% of matched studies agreed with respect to treatment effects (kappa statistic, 0.037), and only 62% of the HRs in observational studies were within the 95% CI of the matched RCT. None of these correlations exceeded what would be expected by chance, and correlations did not improve in the studies that used the most sophisticated statistical methods of analysis, including propensity score weighting, instrumental variable adjustment, or sensitivity analysis. Of note, among the 70 observational studies ranked as rigorous and that reported overall survival, 35 reported a positive result, while the RCT reported either no difference or showed an effect in the opposite direction.

This outcome is arguably the most important one to patients and is also the most easily defined and least subject to investigator bias.

Although this may be the most biologically important outcome in a disease-specific intervention, it is a more subjective end point than total mortality and more subject to investigator bias in its determination. This end point may also miss important effects of therapy that may actually shorten overall survival.

This is an extremely important end point to patients. Careful documentation of this end point within a strong study design is therefore sufficient for most physicians to incorporate a treatment into their practices.

These end points may be subject to investigator interpretation. More importantly, they may, but do not automatically, translate into direct patient benefit such as survival or quality of life. Nevertheless, it is rational in many circumstances to use a treatment that improves these surrogate end points while awaiting a more definitive end point to support its use.