Limitations of Meta-analysis

Improving Medical Statistics www.improvingmedicalstatistics.com
and Interpretation of Clinical Trials
_____________________________________________________

LIMITATIONS OF META-ANALYSES

The assumption that a meta-analysis routinely represents the final and accurate viewpoint in an area of research is not warranted.

A meta-analysis combines similar trials in order to obtain a larger number of patients to improve the evaluation of whether statistically reliable differences exist between comparison groups. Meta-analyses are by no means perfect. On some occasions, a large clinical trial has subsequently been performed evaluating the same clinical question with an outcome quite different from the initial meta-analysis. Discrepancies between meta-analyses and subsequent large randomized clinical trials are documented in literature¹.

The conclusions made by the authors of a meta-analysis are subject to the same potential for bias as the smallest of clinical studies. The authors of the meta-analysis must assess the limitations of their analysis and decide what conclusions to state. In addition, they need to determine how broadly their conclusions can be applied and to what patient groups. Conservative conclusions derived directly from the data with a realistic assessment of the limitations of the study are optimal, but by no means universal. A meta-analysis is particularly subject to biased conclusions when it is created by advocates of a controversial opinion regarding the same topic the meta-analysis is addressing. (See critique of oat bran meta-analysis, critique of ALLHAT meta-analysis, and Beware of Meta-Analyses Bearing False Gifts for details.)

Similarly, a meta-analysis written by employees or representatives of a pharmaceutical company will have an inherent and expected favorable bias towards the product of that company. This type of meta-analysis will always be an advocacy meta-analysis.

A very large randomized clinical trial is the most reliable way of obtaining reproducible results. This means that if the same trial protocol was repeated with a similar patient population using a sufficient number of patients, the same trial results would be expected to occur. However, even a very large trial does not guarantee that the specific treatment protocol being studied has been constructed optimally or appropriate conclusions have been formulated. (See Tale of Two Large Trials.)

The more similar the trials are that are being added together, the more likely the meta-analysis will result in valid conclusions. The addition of study protocols that are significantly different from one another makes a meta-analysis less reliable.

Since a meta-analysis is a summation of trials, it is only as good as the trials that are combined in the meta-analysis. If a very large trial is poorly done and is part of a meta-analysis, the results of the meta-analysis can be adversely impacted by that trial.

A meta-analysis has a number of areas with the potential for bias (which is usually unintentional). The potential areas of bias in a meta-analysis include:

1. Inclusion/exclusion criteria used to select the studies for the meta-analysis.

2. Methods used to perform the meta-analysis.

3. The conclusions which are reached.

4. Statements by the authors regarding the reliability of the results of their meta-analysis.

5. Declarations of broad applicability for the conclusions of a particular meta-analysis.

Meta-analyses can be quite useful and beneficial for the analysis of similar trials. However, the assumption that every meta-analysis represents the final and accurate viewpoint on an area of research is unwarranted.

NETWORK META-ANALYSIS

There is a type of meta-analysis called network meta-analysis that is more subject to erroneous conclusions than a routine meta-analysis. A network meta-analysis adds an additional variable to a meta-analysis. Rather than simply summing up trials that have evaluated the same treatment compared to placebo (or compared to an identical medication), different treatments are compared by inference. ( If A is better than B, and B is equal to C, then A is better than C.)

The problem with network analysis in regards to a meta-analysis is that a network meta-analysis tends only to be valid for very similar studies. Since network meta-analysis combines studies with a higher degree of variability, there is even more potential for combining studies that are not adequately similar. The quality of some recent network analyses in the hypertension literature highlights the problems of this type of analysis.

1. Discrepancies Between Meta-Analyses and Subsequent Large Randomized, Controlled Trials. LeLorier J, Gregoire G, et al. NEJM, 1997; 337:536-42.

Hit Counter